(1) Field of the Invention
The present invention relates to a display control method using image measurement and image recognition, and particularly to a technology which detects a position and an orientation of an apparatus on which a camera is mounted, using an image captured by the camera, and controls a display based on the detected position and orientation of the apparatus.
(2) Description of the Related Art
In recent years, with the wide use of the Internet, the so-called ubiquitous network society is being formed, in which computers and household electrical appliances are interconnected and remote control, media fusion and the like are realized. The society is characterized in having a high degree of flexibility which enables users to perform communications at any time and any place, and the users can act without recognizing differences in apparatuses and the physical distance between the user and an apparatus. Although in the realization of the ubiquitous network society, the network connection is an essential condition for the system, a high degree of flexibility in the user interface between the user and the apparatus is also a prerequisite. Even when an apparatus is connected as either hardware or software, unless the user who uses the system can freely communicate with the system, it is hard to say being ubiquitous. To this end, the technology which enhances ease in using a system, in other words, the technology which enhances usability of the system is required.
In order to realize an intuitive operation method for enhancing the usability of a system, it is necessary for the system to automatically understand the intention of the user. For the realization, for example, a technology which detects positions and orientations of interface devices is useful.
One of such interface devices is a pointing marker. For example, in FIG. 1 of Japanese Laid-Open Patent Application No. 2001-356875 (referred to as Patent Reference 1), a user holds, in a hand as a pointing marker, an object (blue LED and the like) with brightness or color which is different from that of the user. A position of the pointing marker is detected by an image sensor in a system, and in accordance with the detected position, a displayed image can be operated by the system. With this, the user can perform communications with the system intuitively with bodily sensation, without understanding the operation method of the system.
In addition, using brightness of a subject and color difference between the subject and an image of the subject, there exists a method of extracting a marker from a captured image (for example, refer to FIG. 1 of Japanese Laid-Open Patent Application No. 2000-230806 which is referred to as Patent Reference 2 hereinafter). The method is that a current marker position is compared to a reference marker position captured in advance so as to detect a distance between the subject and the image sensor. With the method, the user has only to register the reference marker by executing predetermined initial processing in activating the system. Then, the system detects changes in the position of the marker which are sequentially captured, and automatically obtains the distance between the subject and the image sensor.
In addition, the system, referred to as the virtual reality system, which gives the user sensation as if the user directly operated a real object enables the user to directly control, without recognizing the apparatus, objects in virtual space which has been built using computer graphics. For example, when a portable display which the user holds in the hand is moved three-dimensionally, the system detects the motion by an acceleration sensor and the like, and changes the displayed image to another image according to the motion (for example, refer to FIG. 11 of Japanese Patent Laid-Open Application No. 11-18025 which is referred to as Patent Reference 3 hereinafter). Thus, it is possible to intuitively provide the system with information regarding changes in viewpoints in the virtual reality space by changes in the position and orientation of the display held in the hand.
However, the technology described in Patent Reference 1 requires preparation of a special marker. Thus, there exists a problem in that the general versatility lacks and situations where the technology is used are limited. In other words, as the brightness of a subject and color difference between the subject and an image of the subject are used for detecting the marker, it is necessary to provide a marker with the brightest color among colors of any other subjects and a color which is not included in any other subjects.
In addition, with the technology described in Patent Reference 2, a single image sensor tracks plural markers and detects a distance between a subject and the image sensor. However, even with the technology, it is not possible to detect a three-dimensional position of the image sensor. In other words, as the distance from the center of a sphere to any points on the sphere is all equal, such position information is regarded as the same. Thus, there remains a problem in the usability.
On the other hand, as the acceleration sensor or the like is used in the technology of Patent Reference 3, there is no need to capture the marker. Thus, it outperforms other system configurations in terms of the general versatility. However, for example, as the acceleration sensor can not detect a speed of uniform motion, there exists a problem in terms of sensitivity of the sensor. To overcome this point, sensor fusion is effective which multi-directionally detects position/orientation information by a magnetometric sensor, a ultrasonic sensor or the like so as to complement the information each other. However, an additional sensor leads to increase in the cost of the system and increase in the capacity and weight of an apparatus. In addition, in many cases, such sensors constantly operate, and increase in the number of sensors leads to increase in electric power consumption of the apparatus. In particular, this poses a serious problem to portable apparatuses.
Thus, the present invention has been conceived in view of the aforementioned circumstances, and the object is to provide a display control method and a display control device which can detect a position and an orientation of an apparatus by a simple structure and control a display of a displayed image based on the detected position and orientation of the apparatus.